ON EXISTENCE OF DOUBLE COSET VARIETIES 
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Abstract. Let G be a complex affine algebraic group and H, F C G be closed subgroups. 
The homogeneous space G / H can be equipped with structure of a smooth quasiprojective 
variety. The situation is different for double coset varieties F\\G//H. In this paper we give 
examples showing that the variety F\\G/'H does not necessarily exist. We also address the 
question of existence of F^G/^H in the category of constructible spaces and show that under 
sufficiently general assumptions F\\G//H does exist as a constructible space. 



1. Introduction 

Let G be a complex affine algebraic group and H C G be a closed subgroup. By Chevalley 
Theorem the set of left H-cosets can be equipped with a uniquely defined structure of a 
smooth quasiprojective variety such that G act morphically on G / H. Moreover, the projec- 
tion G — 7- G / H is a geometric quotient for the action of H on G by right multiplication. 

The construction of the homogeneous space G / H has a natural generalisation: one can 
take another subgroup F C G and consider double cosets, i. e. the sets F gH, g & G. These 
cosets are orbits of the action of F x H on G given by the formula (/, h) o g = fgh^^. It 
is clear that such action, unlike the action of H on G by multiplication, can have orbits of 
different dimensions, thus it does not necessarily admit a geometric quotient. Because of 
this we consider a weaker quotient, namely, a categorical one. 

The double coset variety F\\G//H is defined to be the underlying space of the categorical 
quotient G — t- F\\G//II with respect to the described action of F x H, if this quotient exists. 
If the subgroups F and H are reductive then this variety exists and coincides with the 
spectrum Spec(^C[G]^) of the algebra of regular functions on G invariant under the action 
of F X H. Moreover, if G is also reductive then by a result of Luna [9j the action F x H : G 
is stabl^, hence F\\G//H parametrises generic (closed) double cosets. 

In this paper we consider the case when the subgroups F and H are not reductive. In this 
setting one can not guarantee that F\\G//H = Spec(^C[G]^) ; moreover, F\\G//II does not 
necessarily exist. To illustrate this we give the following examples: 

I. A unipotent group G and a subgroup U of G such that the variety U\\G//U does not 
exist. 

II. A reductive group G and two subgroups F, H such that the variety F\\G//H does not 
exist. 
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"'^Reductivity of G is essential: consider the group B of the upper-triangular matrices and its subgroup T 
of the diagonal matrices; the action T x T : B is not stable. 
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III. A semisimple group G and two subgroups F, H such that the algebra of F x H-invariant 
regular functions R = ^C[G]^ is finitely generated and the natural morphism vr : G — > 
Spec-R is surjective, but vr is not a categorical quotient. 
It is interesting to remark that though U\\G//U considered in Example [T] does not exist as 
an algebraic variety, it does exist as a constructible space. Thus, here we observe the same 
phenomenon as in j2], [1] and [5], namely, an action that admits no quotient in the category 
of algebraic varieties does admit one in the category of constructible spaces. 

In Example IIIII the categorical quotient F\\G//II exists in the category of algebraic vari- 
eties; its underlying space is the blow-up of Spec(^C[G]^) at one point; moreover, in this 
example the categorical quotient separates generic double cosets. 

The author would like to thank I. V. Arzhantsev for stating the problem and helpful 
discussions. 

2. Preliminaries on categorical quotients 

Let an algebraic group G act on an algebraic variety X. Recall that the categorical 
quotient of this action is a G-invariant (i. e., constant on G-orbits) morphism ttq : X ^ Y 
such that every G-invariant morphism if : X ^ Z factors uniquely through tiq, that is, there 
is a unique morphism (f making the following diagram commutative: 




The universal property of ttg implies that Y is defined uniquely up to isomorphism. Remark 
that ttq is necessarily surjective. By abuse of language we will sometimes call the variety Y = 
Xjj G the categorical quotient. 

If G is reductive and X is affine then the categorical quotient for the action G : X 
is ttg : X — )■ y = SpecC[X]'~^ with morphism ttq corresponding to inclusion C[X]*^ C C[X]; 
in this case ttq has an important additional property: it separates closed orbits. If G is 
not reductive then the quotient Xjj G does not necessarily exist. Examples of actions not 
admitting a categorical quotient are given in [TT| 4.3], [1], [2]. Let us point out one example 
that we will make use of. 

Example 2.1. |TT| 4.3] There is no categorical quotient for the action of a one-dimensional 
unipotent group U on space Mat2x2 of 2 x 2-matrices given by the formula 

A o ' ' — ' 



0-21 022 J V 1 J \0'21 O22 

Remark that we have C[Mat2x2]'^ = C[a2i, 022, det] and the canonical morphism n : Mat2x2 
Spec C[Mat2x2]^ — separates U-orbits of generic points having 021 7^ or 022 7^ 0. The 
image of this morphism is A'^ without the punctured line {021 = 022 = 0, det 7^ 0}. Since 
the image of it is not open, by j2l Corollary 1.4] the action U : Mat2x2 has no categorical 
quotient in the category of algebraic varieties. 

The morphism vr considered in Example 12.11 can be regarded as a quotient morphism 
after an appropriate modification to the definition of categorical quotient. It turns out 
that admitting only morphisms into varieties as categorical quotients is overly restrictive for 
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certain actions G : X. To work around this Bialynicki-Birula introduced in |3] the category of 
dense constructible subsets. This approach has been further developed in [2] to permit maps 
into constructible spaces as candidates for quotient morphisms. Recall that a constructible 
space is a topological space with a sheaf of functions admitting a finite cover by subsets 
that are isomorphic (as spaces with functions) to constructibe subsets of affine varieties. A 
morphism of constructible spaces is a morphism of spaces with functions. We say that a 
constructible quotient is a categorical quotient in the category of constructible spaces. It is 
possible for an action G : X to have no quotient in the category of algebraic varieties, but 
to have a constructible quotient. 

Example 2.2. Let a unipotent group G act on a vector space V. It follows from |21 Corol- 
lary 1.2] that the action G : V admits a constructible quotient, provided that C[V^]*^ is 
finitely generated, li p : V ^ SpecC[V]'^ is the morphism corresponding to the inclu- 
sion C[^]*~' C C[V] then the constructible quotient is p : — p{V). In particular, the 
map TT in Example 12.11 is a constructible quotient for the action U : Mat2x2- 

Let us point out a fact concerning quotients under two commuting actions; it will be used 
to identify double coset varieties with quotients of homogeneous spaces. Let F x H act on a 
variety X and let there be a categorical quotient iTp : X ^ Y = Xjj F. The group H acts 
on Y as an abstract group: '\i y = vrF(x) then h o y = TTp[h o x). By [Si Theorem. 7.1.4] this 
action is regular. Moreover, existence of Y// H is equivalent to existence of X// (F x H) and 
these two quotients coincide: 

X > X//(F X H) = F// H . 

Y = X//F 

The following statement will be used in Proposition O 

Lemma 2.3. Let an algebraic group G act on an algebraic variety Y . Suppose that there 
is yo & Y that belongs to closure of every G-orbit. Consider the action G : X x Y , 
where X is a normal variety and G acts trivially on the first factor. Let W X x Y be a 
G-invariant open subset. Suppose that W contains Xq x Y , where Xq C X is a dense subset, 
and pr(l^) = X , where pr is the projection onto the first factor. Then the action G : W 
has pi : W ^ X as a categorical quotient both in the category of algebraic varieties and in 
the category of constructible spaces. 

Proof. Let us fix a G-invariant morphism (p : W ^ Z into an algebraic variety Z (resp., into 
a constructible space) and show that it factors uniquely through pr. 

Step 1. We claim that (p extends to a continuous map on W U X x {yo}. Let us fix a 
point {x'^yo) ^ X x {yo] and an arbitrary sequence C Xq such that Xn — )■ x' . Now 

we show that the sequence (f{xn,yo) converges. Since pr(Vr) = X, there is a point {x',y) 
in W for some y &Y. The points (x„,yo) and {xn,y) belong to W, hence by G-invariance 
of ip we have ip{xn,y) = ip{xn,yo), thus lim ip{xn,yo) = lim 'p{xn,y) = v{x',y). Since 

n— >+oo n— >+oo 

a converging sequence can have only one limit, lim if{xn,yo) does not depend on choice 

of (x', y) G W. For the extended map (p we have ip{x, y) = ip{x, yo), so continuity of ip\xx{yo} 
implies continuity of (p on W U X x {yo}. 
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Step 2. Now we show that X x {yo} can be covered by open affine sets Xi x {yo} such 
that the image of (f : Xi x {yo} — )• Z is contained in some affine subset of Z. Let {Zi} be 
an affine covering of Z and {Ui} be an affine covering of X. The set Vij = (f~^{(p{Ui) fl Zj) 
is open in Ui. Every set Vij is a union of principal open subsets Vij = UkVijk- The sets Vijk 
make up the required covering of X x {yo}. 

Step 3. Since ip{x,y) = ip{x,yo), we have ip = (p o where p = p\xx{yo} and p is 
the map W ^ X x {yo}, pix,y) = {x,yo). Denote z the identification of X and X x {yo}: 
t{x) = {x, yo). We have p> = <^o^o7^, so p factors through vr. It remains to verify that p\xx{yo} 
is a morphism. The variety X is normal, hence the affine opens Xi x {yo} constructed at 
step 2 are normal varieties, too. Restrictions of p to these opens are morphisms of affine 
varieties; if Z is a constructible space then p\xix{yo} is a morphism into a constructible set, 
but it can be regarded as a morphism into an affine variety containing p{Xi x {yo})- By 
theorem on removable singularities the continuous extensions of p\x,x{yo} are morphisms. □ 

3. Existence and non-existence of double coset varieties 

[31.1. Consider a unipotent group G and a subgroup U: 

* * *\ /l * o\ 

1 * * 10 

10' 10' 

V V V 

here * denotes an arbitrary number. We claim that if we take F = H = U then the double 
coset variety F\\G//H does not exist. Remark that the group FxH = UxUis unipotent, 
hence every double (U, U)-coset is closed [IT, 1.3]; had F and H been reductive, this would 
have implied existence of the geometric quotient G — )■ G/(F x H). 

Proposition 1. The action U x U : G has no categorical quotient in the category of alge- 
braic varieties. It admits a constructible quotient and the constructible quotient parametrises 
generic double cosets. 

Proof. Consider the action of G on space of 4 x 2-matrices by left multiplication. The 
subgroup U is the stabiliser of the matrix 



/o 


o\ 








1 





^0 


V 



Therefore, the homogeneous space G / U is isomorphic to and can be identified with 
the variety of matrices 







* 

1 


* 



^0 


V 



After this identification the action of U on G / U becomes the matrix multiplication; it is 
therefore isomorphic to the action of U on space Mat2x2 of 2 x 2-matrices by left multiplica- 
tion. Example 12.11 shows that this action does not admit a categorical quotient. Therefore, 
U \\G// U = (G / U)// U does not exist. 
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/I 



G 



From Example 12.21 it follows that the action U x U : G has a constructible quotient 
TT : G — 7- U\\G// U C which separates generic double cosets. □ 



Remark. The constructible quotient vr : G — t- U\\G//U does not separate all closed double 
cosets. Indeed, all 2 x 2-matrices with = 022 = (we use notation of Example 12. ip are 
fixed under the action of U and have det = 0, hence their preimages in G are closed (U, U)- 
cosets, which are mapped by vr to G A'^. 

[31,2. Take G = GL4 and consider the action of G on 4 x 2-matrices. Let H be the 
stabiliser of the matrix M, M being the same as in 13.11 The homogeneous space W = G / Yi 
is identified with the variety of 4 x 2-matrices with non-zero columns. Let F be the subgroup 
of G consisting of the following matrices: 



/I 



v 



a 
1 



0\ 






a e 



s e 



The subgroup F acts on W via matrix multiplication. 

Proposition 2. The action F x H : G does not admit a categorical quotient in the category 
of algebraic varieties, but has a constructible quotient. 

Proof. The group F is a direct product F = U x S of one-dimensional unipotent group U and 
one- dimensional torus S. The categorical quotient for the action S : W is pi : W —> Mat2x2; 
where the projection pr erases the bottom half of matrices of W. Indeed, one can apply 
Lemma 12.31 with the acting group S and X = Y = Mat2x2 representing top and bottom 
halves of matrices respectively, and with Xq consisting of matrices with non-zero columns. 

Thus, had the quotient W// F = F\\GL4//H existed, it would have been also (W// S)// U = 
Mat2x2 // U, but, according to Example 12. ![ the latter quotient does not exist. 

By Lemma 12.31 and Example 12. 2[ the actions S : W and U : Mat2x2 both have a con- 
structible quotient, thus F\\GL4//H = {W// S)//U exists as a constructible space. □ 

[31,3. This example is based on ^ 4.5]. Consider the following symmetric bilinear form 
on C'^: (ci, 64) = (e2, 63) = 1 and the other pairings of basis vectors are zero. The cone 

X = {x G I X1X4 + X2X3 = 0} \ {(0, 0, 0, 0)} 

is the collection of non-zero isotropic vectors, therefore X = SO4 / H, where H is the stabiliser 
of a non-zero isotropic vector. As F we take the following unipotent subgroup of SO4: 

/l a \ 




a 
1 



v 



a G C. 



The algebra C[X]^ is freely generated by X2 and X4; indeed, these two functions are F- 
invariant and generic orbits meet the plane {xi = X3 = 0}, so there are no other generators. 



It is clear that the canonical morphism vr : X — )■ SpecC[X] 



is surjective. Neverthe- 



less, SpecC[X]^ is not the quotient for F : X because the following morphism ip : X 
does not factor through vr: 



!f{Xi,X2,X3,Xi) 



{X2 : X4) 
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(xi : -X3). 



Indeed, 7r(a;i, X2, X3, X4) = {x2,X4), and from if = (p o n it follows that <f{x2,X4) = {x2 '■ X4) 
when X2 7^ or X4 7^ 0, hence if is not continuous in (0, 0), which is not possible. 

Let us show that the considered action has a categorical quotient, though it does not 
coincide with SpecC[X]^. 

Proposition 3. The action F : X has a categorical quotient in the category of algebraic 
varieties. 

Proof. The quotient is the blow-up of the origin in 

= {((x, y),iu: v)) e A'^ xF^ \ xv-yu = 0} 

with the morphism p : X — A^, p{xi,X2, X3, X4) = ((x2, ^4), (xi : —X3)) = {{x2, X4), {x2 : X4)). 

Let us check that every F-invariant morphism ip : X Z factors through p. By F- 
invarience of ip we have ^9(0x1, X2, CX3, X4) = X2, 0:3, X4). Indeed, if one of X2,X4 is not 
zero then the point (xi, 0:2, 0:3, X4) and the points (cxi, X2, CX3, X4) belong to one orbit of F; 
if X2 = 0:4 = then we have 

V9(Xi,0,X3,0) = lim(^(xi,txi,X3, -txg) = 

= lim(/?(cxi, txi, CX3, — txs) = ificxi, 0, 0x3, 0). 

Define if : A"^ —> Z as the morphism taking {{x,y), {u : v)) to ip{u,x, —v,y). The reasoning 
above shows that ip is well defined. Thus, p = p o i. e., p factors through p. Since 
p{X) = A^, the morphism p can be chosen uniquely. □ 

Remark. It is clear that p : X — > A^ separates orbits of points having X2 7^ or X4 7^ 0. How- 
ever, p does not separate all closed orbits: the points z = (xi, 0, X3, 0) and z' = (cxi, 0, CX3, 0) 
are F-fixed, but p{z) = p{z'). Thus, the quotient q : SO4 — ?■ F\\S04//II = A^ separates 
generic double cosets, but fails to separate all closed double cosets. 

[31,4. Remark that in Examples HI and [ITl the actions F x H : G have no categorical quotient 
in the category of algebraic varieties but do admit one in the category of constructible spaces. 

Question. Let G be a connected affine algebraic group and F, H be closed subgroups in G. 
Is it true that F\\G//II exists as a constructible space? 

The following proposition gives a partial answer to this question. 

Proposition 4. Let G be a connected affine algebraic group and F, H C G &e closed connected 
subgroups with trivial character groups. Suppose that the algebra ^C[G]^ is finitely generated 
and let IT : G ^ Spec(^C[G]^) be the canonical morphism. Then F\\G//II exists as a 
constructible space and the map tt : G — )■ 7r(G) is the constructible quotient for the action 
o/FxH on G. 

Proof. By [10^ Theorem 6], the underlying variety of G has a finite divisor class group. 
Additionally, F and H have trivial character groups, therefore every F x H-invariant hyper- 
surface D C G is the zero set of an invariant function //^ G ^C[G]^. It follows from |21 
Corollary L2] that the action F x H : G has vr : G — ?■ 7r(G) as a constructible quotient. □ 

Remark. One can often give a positive answer to the question on finite generation of ^C[G]^. 
Recall that if R is a reductive group, Z is an affine R-variety and U C R is a maximal 
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unipotent subgroup then the algebra C[Z]^ is finitely generated [H Chapter 3.2]. Thus, 
the constructible space F\\G// H is guaranteed to exist if both groups F and H are maximal 
unipotent subgroups in bigger reductive subgroups F', H' C G or if one of them is semisimple 
and the other one is a maximal unipotent subgroup in a bigger reductive subgroup. Other 
results on finite generation of algebras of invariants can be found in [B]. 
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